Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from pydata:main #653

Merged
merged 3 commits into from
Feb 10, 2025
Merged

[pull] main from pydata:main #653

merged 3 commits into from
Feb 10, 2025

Conversation

pull[bot]
Copy link

@pull pull bot commented Feb 10, 2025

See Commits and Changes for more details.


Created by pull[bot] (v2.0.0-alpha.1)

Can you help keep this open source service alive? 💖 Please sponsor : )

max-sixty and others added 3 commits February 9, 2025 21:12
* Start splitting up `dataset.py`

Currently, `dataset.py` is 10956 lines long. This makes doing any work with current LLMs basically impossible — with Claude's tokenizer, the file is 104K tokens, or >2.5x the size of the _per-minute_ rate limit for basic accounts. Most of xarray touches it in some way, so you generally want to give it the file for context.

Even if you don't think "LLMs are the future, let's code with vibes!", the file is still really long; can be difficult to navigate (though OTOH it can be easy to just grep, to be fair...).

So I would propose:
- We start breaking it up, while also being cognizant that big changes can cause merge conflicts
- Start with the low-hanging fruit
  - For example, this PR moves code outside the class (but that's quite limited)
  - Then move some of the code from the big methods into functions in other files, like `curve_fit`
- Possibly (has tradeoffs; needs discussion) build some mixins so we can split up the class, if we want to have much smaller files
- We can also think about other files: `dataarray.py` is 7.5K lines. The tests are also huge (`test_dataset` is 7.5K lines), but unlike with the library code, we can copy out & in chunks of tests when developing.

(Note that I don't have any strong views on exactly what code should go in which file; I made a quick guess — very open to any suggestions; also easy to change later, particularly since this code doesn't change much so is less likely to cause conflicts)

* .
Seems easy locally; maybe too easy and something will break in CI...
* add kwargs to map_over_datasets (similar to apply_ufunc), add test.

* try to fix typing

* improve typing and simplify kwargs-handling per review suggestions

* apply changes to DataTree.map_over_datasets

* add whats-new.rst entry

* Update xarray/core/datatree_mapping.py

Co-authored-by: Mathias Hauser <[email protected]>

* add suggestions from review.

---------

Co-authored-by: Mathias Hauser <[email protected]>
@pull pull bot added the ⤵️ pull label Feb 10, 2025
@pull pull bot merged commit 1189240 into Illviljan:main Feb 10, 2025
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants